Augmented reality bone landmark display

ABSTRACT

A method or system for using an augmented reality device may include displaying information in a surgical field. A method may include receiving an indication of a location of a landmark on a bone of a patient, retrieving a planned location of the landmark on the bone of the patient and receiving information corresponding to the location or the planned location. The location or the planned location may be displayed as a virtual indication using an augmented reality display of the augmented reality device, for example while permitting the surgical field to be viewed through the augmented reality display.

CLAIM OF PRIORITY

This application claims the benefit of priority to U.S. ProvisionalApplication No. 63/052,137 filed Jul. 15, 2020, titled “INSTRUMENTPREPARATION AND VALIDATION,” which is hereby incorporated herein byreference in its entirety.

BACKGROUND

Surgical advancements have allowed surgeons to use preoperativeplanning, display devices within a surgical field, optical imaging, andguides to improve surgical outcomes and customize surgery for a patient.While these advances have allowed for quicker and more successfulsurgeries, they ultimately rely on physical objects, which have costsand time requirements for manufacturing and configuration. Physicalobjects and devices may also obstruct portions of a surgical field,detracting from their benefits.

Computer-assisted surgery is a growing field that encompasses a widerange of devices, uses, procedures, and computing techniques, such assurgical navigation, pre-operative planning, and various robotictechniques. In computer-assisted surgery procedures, a robotic systemmay be used in some surgical procedures, such as orthopedic procedures,to aid a surgeon in completing the procedures more accurately, quicker,or with less fatigue.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsmay describe similar components in different views. Like numerals havingdifferent letter suffixes may represent different instances of similarcomponents. The drawings illustrate generally, by way of example, butnot by way of limitation, various embodiments discussed in the presentdocument.

FIG. 1 illustrates surgical field in accordance with some embodiments.

FIG. 2 illustrates an AR instrument identification display in accordancewith some embodiments.

FIG. 3 illustrates a system for displaying virtual representations of alandmark in accordance with some embodiments.

FIG. 4 illustrates a flowchart showing a technique for displayingvirtual representations of a landmark in accordance with someembodiments.

FIG. 5 illustrates a surgical field including a virtual representationof a remote surgical field, for example for use with an augmentedreality display in accordance with some embodiments.

FIG. 6 illustrates a flowchart showing a technique for displaying avirtual representation of a remote surgical field within a localsurgical field in accordance with some embodiments.

FIG. 7 illustrates a robot sterilization system in accordance with someembodiments.

FIG. 8 illustrates a flowchart showing a technique for storing asterilized instrument using a robotic system in accordance with someembodiments.

FIG. 9 illustrates a system for surgical instrument identification usingan augmented reality display in accordance with some embodiments.

FIG. 10 illustrates generally an example of a block diagram of a machineupon which any one or more of the techniques discussed herein mayperform in accordance with some embodiments.

FIGS. 11A-11B illustrate user interface components for landmark planningand plan evaluation in accordance with some embodiments.

DETAILED DESCRIPTION

Systems and methods for using an augmented reality device during asurgical procedure are described herein. The systems and methods hereindescribe uses for the augmented reality device, such as to display alandmark or representations of real objects overlaid on a realenvironment. An augmented reality (AR) device allows a user to viewdisplayed virtual objects that appear to be projected into the realenvironment, which is also visible. AR devices typically include twodisplay lenses or screens, including one for each eye of a user. Lightis permitted to pass through the two display lenses such that aspects ofthe real environment are visible while also projecting light to makevirtual elements visible to the user of the AR device.

FIG. 1 illustrates surgical field 100 in accordance with someembodiments. The surgical field 100 is illustrated in FIG. 1 including asurgeon 102, a patient 108, and may include a camera 112. The surgeon102 is wearing an augmented reality (AR) device 104 which may be used todisplay a virtual object 110 to the surgeon 102. The virtual object 110may not be visible to others within the surgical field 100 (e.g.,surgical assistant 114 or nurse 120), though they may wear AR devices116 and 122 respectively. Even if another person is viewing the surgicalfield 100 with an AR device, the person may not be able to see thevirtual object 110 or may be able to see the virtual object 110 in ashared augmented reality with the surgeon 102, or may be able to see amodified version of the virtual object 110 (e.g., according tocustomizations unique to the surgeon 102 or the person) or may seedifferent virtual objects entirely. Augmented reality is explained inmore detail below.

Augmented reality is a technology for displaying virtual or “augmented”objects or visual effects overlaid on a real environment. The realenvironment may include a room or specific area (e.g., the surgicalfield 100), or may be more general to include the world at large. Thevirtual aspects overlaid on the real environment may be represented asanchored or in a set position relative to one or more aspects of thereal environment. For example, the virtual object 110 may be configuredto appear to be resting on a table. An AR system may present virtualaspects that are fixed to a real object without regard to a perspectiveof a viewer or viewers of the AR system (e.g., the surgeon 102). Forexample, the virtual object 110 may exist in a room, visible to a viewerof the AR system within the room and not visible to a viewer of the ARsystem outside the room. The virtual object 110 in the room may bedisplayed to the viewer outside the room when the viewer enters theroom. In this example, the room may act as a real object that thevirtual object 110 is fixed to in the AR system.

The AR device 104 may include one or more screens, such as a singlescreen or two screens (e.g., one per eye of a user). The screens mayallow light to pass through the screens such that aspects of the realenvironment are visible while displaying the virtual object 110. Thevirtual object 110 may be made visible to the surgeon 102 by projectinglight. The virtual object 110 may appear to have a degree oftransparency or may be opaque (i.e., blocking aspects of the realenvironment).

An AR system may be viewable to one or more viewers, and may includedifferences among views available for the one or more viewers whileretaining some aspects as universal among the views. For example, aheads-up display may change between two views while virtual objects maybe fixed to a real object or area in both views. Aspects such as a colorof an object, lighting, or other changes may be made among the viewswithout changing a fixed position of at least one virtual object.

A user may see the virtual object 110 presented in an AR system asopaque or as including some level of transparency. In an example, theuser may interact with the virtual object 110, such as by moving thevirtual object 110 from a first position to a second position. Forexample, the user may move an object with his or her hand. This may bedone in the AR system virtually by determining that the hand has movedinto a position coincident or adjacent to the object (e.g., using one ormore cameras, which may be mounted on an AR device, such as AR devicecamera 106 or separate, and which may be static or may be controlled tomove), and causing the object to move in response. Virtual aspects mayinclude virtual representations of real world objects or may includevisual effects, such as lighting effects, etc. The AR system may includerules to govern the behavior of virtual objects, such as subjecting avirtual object to gravity or friction, or may include other predefinedrules that defy real world physical constraints (e.g., floating objects,perpetual motion, etc.). An AR device 104 may include a camera 106 onthe AR device 104 (not to be confused with the camera 112, separate fromthe AR device 104). The AR device camera 106 or the camera 112 mayinclude an infrared camera, an infrared filter, a visible light filter,a plurality of cameras, a depth camera, etc. The AR device 104 mayproject virtual items over a representation of a real environment, whichmay be viewed by a user.

Eye tracking may be used with an AR system to determine which instrumenta surgeon wants next by tracking the surgeon's eye to the instrument. Inan example, a nurse or surgical assistant may then retrieve thedetermined instrument. The determined instrument may be presented in ARto the nurse or surgical assistant. In another example, the surgeon mayspeak the instrument (e.g., using a pre-selected code word, using speechprocessing and word recognition, via saying a number, or the like). Thevoice command may be combined with eye tracking, in still anotherexample, to find an instrument;

The AR device 104 may be used in the surgical field 100 during asurgical procedure, for example performed by the surgeon 102 on thepatient 108. The AR device 104 may project or display virtual objects,such as the virtual object 110 during the surgical procedure to augmentthe surgeon's vision. The surgeon 102 may control the virtual object 110using the AR device 104, a remote controller for the AR device 104, orby interacting with the virtual object 110 (e.g., using a hand to“interact” with the virtual object 110 or a gesture recognized by thecamera 106 of the AR device 104). The virtual object 108 may augment asurgical tool. For example, the virtual object 110 may appear (to thesurgeon 102 viewing the virtual object 110 through the AR device 104) asa representation of a landmark previously placed on a patient bone. Inanother example, the virtual object 110 may be used to represent aplanned location of a landmark (e.g., using a pre-operative image and acaptured image of the bone in the real space). In certain examples, thevirtual object 110 may react to movements of other virtual or real-worldobjects in the surgical field. For example, the virtual object 110 maybe altered by a to move a landmark (e.g., a placed landmark). Furtherdiscussion of virtual landmarks is discussed below with respect to FIGS.3-4.

In other examples, the virtual object 110 may be a virtualrepresentation of a remote surgical field (e.g., an entire OR, a camerafield of view of a room, a close-up view of a surgical theater, etc.).In this example, the virtual object 110 may include a plurality ofvirtual objects. Further discussion of this example is provided belowwith respect to FIGS. 5-6.

FIG. 2 illustrates an augmented reality (AR) instrument identificationdisplay 200 in accordance with some embodiments. Prior to any surgicalprocedure, the nursing staff unloads trays, and prepares and placesinstrumentation for the procedure on a table. This process may befastidious and error prone (e.g., missing instrument, misplacement ofinstrument, etc.). A surgeon may have preferences for instrumentplacement, table location, or the like. For example, the table may bepreferred in a particular setup, which may increase consistency andefficiency by removing risks of the wrong tool being picked up, whichmay delay a surgery. Errors due to human choice, staff change, turnover,or the like may be responsible for decreases in efficiency. Theinstrumentation placement process may include a check-list, which istime consuming and also error prone.

The present systems and methods may include a technological solution toerrors in instrument placement by leveraging artificial intelligence oraugmented reality (AR) to ensure correct placement of instruments. Thesystems and methods described herein may tell staff which instrument toplace in what location on a table, for example based on surgeonpreference (e.g., using AR). The systems and methods described hereinmay be used to verify that one or all instruments are correctly placedon the table, such as using an automatic check list verification. In anexample, complicated instruments may be assembled using the systems andmethods described herein.

The benefits of using the present systems and methods include a fasterpreparation or setup of a procedure room (e.g., operating room),eliminating instrument misplacement (improving workflow, efficiency,etc.), and helping avoid the need for surgeon oversight in the process.

The AR instrument identification display 200 includes a surgicalinstrument 206, a virtual indicator 208, and may include additionalinformation 210, such as patient or procedure information. The virtualindicator 208 may be used to identify the surgical instrument 206 thatcorresponds to a procedure being performed. The virtual indicator 208may include moving lights, flashing lights, color or changing colorlights, or other virtual effects. The additional information 210 may forexample, name or provide other information about the surgical instrument206. The virtual indicator 208 may be added to the AR display 200B inresponse to a surgeon selection identifying a need for the surgicalinstrument 206. In an example, when the surgical instrument 206 is orhas been moved, selected, or the surgical assistant otherwise indicatesthat it has been located or identified (or if the surgeon indicates itis no longer needed), the virtual indicator 208 may be removed from theAR display 200. In an example a virtual indicator 212 may be used toidentify an item, such as a correctly or an incorrectly placedinstrument, a verified instrument, or an unknown instrument. A user ofthe AR device used to present the AR display 200 may interact with thevirtual indicator 208, for example by placing a finger, hand, or itemadjacent to or appearing to occupy the same space as the virtualindicator 208. In response, the virtual indicator 208 may perform anaction, such as displaying information about the item represented by thevirtual indicator 208 (e.g., a name of the item, whether the item is aone-time use item or can be re-sterilized, whether the item is fragile,whether the item is a patient-specific or personalized item, whatprocedure the item is to be used for, or the like).

In an example, a schedule for procedures during a day in an operatingroom may be obtained or retrieved by a device. The device may provide ARcapabilities to a user, including instructions for setting up a nextprocedure in the schedule. The users, with the aid of the AR, may placethe instruments in correct position or orientation on a table in theoperating room. After placement of an instrument, a verification processmay be performed, and an output (e.g., correctly placed or incorrectlyplaced, such as with additional instructions for correct placement) maybe provided to the user (e.g., via the AR). When the process iscomplete, and all instruments have been checked as correctly placed bythe verification process, a picture may be taken and a full verificationprocess may be performed to validate the operating room for the givenprocedure. The full verification process may include a second check ofeach instrument, a check of the instruments against needed instrumentsfor the given procedure, timing verification based on the schedule, orthe like. Data may be collected about a surgical procedure, such as atime-series of data based on progression through the procedure, whatsteps occur at what times (e.g., when started or completed), locationsof team members (e.g., surgeon, nurse, etc.) throughout the procedure,camera stills or video of the procedure at various moments, instrumenttracking or use, or the like.

FIG. 3 illustrates a system for displaying virtual representations of alandmark in accordance with some embodiments.

In an example, a landmark may be obtained, such as on a bone of apatient. An AR device may show a virtual representation of the landmarkthat was acquired in a display view 300. The virtual representation maybe displayed on a bone (e.g., a femur 306 or a tibia 308) of the patient(e.g., overlaid on the real bone). The AR device may requestconfirmation (e.g., via a display) to confirm the landmark's location.In an example, a voice command may be used to control the landmarkconfirmation or display with the AR device.

The virtual representations may include representations of surgeongenerated (e.g., selected or registered) landmarks (e.g., 302A, 302B,and 302C) or planned landmarks (e.g., 304A, 304B, and 304C). The ARdisplay view 300 allows the femur 306 and the tibia 308 to be visiblewhile also presenting virtual representations of landmarks. In otherexamples, different bones (e.g., hip, shoulder, spine, etc.) may beviewable. In still other examples, a virtual representation of a bonemay be displayed with the virtual representations of landmarks (e.g.,entirely virtual).

The surgeon generated landmarks may include a landmark 302A, which isdisplayed on the femur 306 separated by some distance from acorresponding planned landmark 304A. The planned landmark 304A may begenerated based on pre-operative planning, for example using a 3D model,an image of the patient, or the like. The planned landmark 304A may beregistered in the real space. For example, a known image or modelcoordinate system may be converted to a coordinate system in the realspace using image processing. The image processing may compare capturedimages of a bone (e.g., in real-time), the patient, a reference object,or the like in real space to previously captured images or a previouslygenerated model. Based on the comparison, a location of the plannedlandmark 304A may be registered on the real femur 306. From thisregistration, further processing may be used to determine how to presenta virtual representation of the planned landmark 304A in the real spacevia an AR display device (e.g., overlaid virtually in the real spacewithin the display view 300).

The surgeon generated landmark 302A may be registered based on an inputdevice (e.g., a pointer that may be used to identify landmarks) or maybe identified directly via the AR device (e.g., with visual processingof an indicated landmark). When using an input device, the registrationto the real space for display in augmented reality may be accomplishedsimilarly to the planned landmarks. In the case where the AR device isused to capture landmark locations directly, the location relative tothe real space is known from the registration process.

The display view 300 may display only virtual representations of surgeongenerated landmarks in one example, only virtual representations ofplanned landmarks in another example, or both in a third example. In thefirst example, the AR device may query the surgeon to confirm theplacements (e.g., audibly, visually, etc.). In the second example, thesurgeon may select virtually represented planned landmarks in the realspace as surgeon generated landmarks. Said another way, the plannedlandmark 304A may be selected to be converted to a surgeon generatedlandmark, in an example. In the third example, the surgeon may bepresented with an option, such as to confirm the surgeon generatedlandmark 302A (e.g., overriding a warning that the surgeon generatedlandmark 302A is some distance from the planned landmark 304A), changingthe landmark location from the surgeon generated landmark 304A to theplanned landmark 304A, re-doing the surgeon generated landmark 304Abased on the identified distance, moving the surgeon generated landmark304A in the direction of the planned landmark 304A (e.g., along a lineor plane, or via freehand movement, such as a gesture visible within thedisplay view 300), or the like.

The landmarks, such as 302C and 304C that are overlapping, at the sameplace, substantially co-located, or adjacent, may be confirmed with asingle entry on a virtual user interface, via a gesture, audibly, etc.,or may be skipped (e.g., not asked to confirm) and assumed to becorrect. A threshold distance for different treatment may be used, andthe threshold distance may be personalized, in an example. The landmarks302B and 304B may greater than the threshold distance in some examples,but less than the threshold distance in some other examples. In someexamples, only landmarks that have a distance between surgeon generatedand planned greater than the threshold may trigger a warning or requireconfirmation input from the surgeon.

In an example, the surgeon generated landmarks may be obtained using arobotic arm, which may include an automated process, a force-assistprocess, a force-resist process, or the like. Even though theselandmarks are referred to herein as surgeon generated, they may beobtained autonomously by the robotic arm. When using the robotic arm,the registration may leverage the coordinate system of the robotic armto translate the landmarks to the display view 300 of the AR device(e.g., rather than or in addition to using image processing or someother technique).

A virtual navigation menu may be presented within the display view 300.The virtual navigation menu may be used to operate aspects of therobotic arm, toggle display of landmarks, proceed to a next step in asurgical procedure, or the like. The navigation menu may be moved orresized within the display view 300, in an example. Movement may occurin response to a gesture, audible instruction, or the like. In anexample, the virtual navigation menu may automatically and virtuallyfollow the robotic arm moving in real space, such as within the displayview 300.

FIG. 4 illustrates a flowchart showing a technique 400 for displayingvirtual representations of a landmark in accordance with someembodiments. The technique 400 may be performed by a processor, forexample by executing instructions stored in memory.

The technique 400 includes an operation 402 to receive an indication ofa location of a landmark on a bone of a patient. The indication may bestored in a database or received directly from a landmark generationdevice (e.g., a pointer). The technique 400 may include registering thebone using a 3D model before receiving the indication of the landmark.For example, a preliminary registration to the 3D model may be performedusing a surface mapping technique via a camera on the AR device or robotarm, subject to confirmation by the landmark registration process. Thepreliminary registration may be inaccurate, but helpful for use duringlandmark registration by orienting the AR device to display virtuallandmarks. A position or orientation of the bone may be determined usingbone tracking, such as via a passive robotic arm.

The technique 400 includes an operation 404 to retrieve a plannedlocation of the landmark on the bone of the patient. The plannedlocation may be retrieved based on a pre-operative image of the bone ofthe patient. The pre-operative image may be registered to a currentpatient space, in an example.

The technique 400 includes an operation 406 to present, using anaugmented reality display, a virtual indication of the landmark at thelocation or the planned location, or both. The virtual indication may bepresented within a surgical field while permitting the surgical field tobe viewed through the augmented reality display.

The technique 400 includes an operation 408 to receive an input relatedto the landmark. The input may include a response to a request forconfirmation of the location of the landmark. Operation 408 may includemoving the location, confirming the location, indicating that thelocation is to be re-selected, validating the location, temporarilyaccepting or denying the location, an indication to remove the virtualindication (which may then be removed), or the like.

In an example, the technique 400 may include using earlier landmarks toiteratively update an expectation of where subsequent planned landmarksare to be identified in a 3D coordinate system of the patient. Forexample, once a few points are registered on the 3D model, the technique400 may include determining that remaining landmarks will accuratelymatch the corresponding points on the 3D model. Iterative updates toorientation or position of the 3D model (e.g., in an AR view) may beperformed based on received landmarks to improve accuracy of the 3Dmodel.

The technique 400 may include displaying a virtual navigation menu inthe augmented reality display. A user may virtually interact with thevirtual navigation menu as if it was displayed on a screen. Anindication may be received to move the virtual navigation menu presentedin the augmented reality display, for example to make the location moreconvenient. The technique 400 may include displaying a live video, usingthe augmented reality display, of the bone using a camera affixed to anend effector of a robotic arm.

FIG. 5 illustrates a surgical field including a virtual representationof a remote surgical field, for example for use with an augmentedreality display in accordance with some embodiments. The surgical fieldmay be viewable within a display view 500 of an AR device. The AR devicemay show a virtual representation of the remote surgical field. In anexample, a voice command or gesture may be used to control whether theremote surgical field is viewable or not.

The display view 500 may be configured to display aspects of the remotesurgical field, such as a remote patient 508 or a remote robotic arm506, displayed in full or zoomed in, such as according to surgeonpreference or control. For example, the display view 500 may include aclose-up view of a leg or bone of the remote patient 508, for exampleduring a surgical procedure.

The display view 500 presents a virtual representation of an aspect ofthe remote surgical field while permitting a local real surgical fieldto be displayed. The real surgical field may include a patient 504 or arobotic arm 502, in some examples. The virtual representation may bedisplayed adjacent to the patient 504, the robotic arm 502, or elsewherewithin the local real surgical field. Adjacent in this context mayinclude separated by an absolute distance within the surgical field,separated by a perceived distance (e.g., appearing in the display view500 to be separated by a foot, a few feet, etc.), anchored in a location(e.g., virtually displayed at a real location within the local surgicalfield), or moved according to surgeon preference. In some examples, thevirtual representation may move when zoomed in or out. For example, whenonly a leg of the remote patient 508 is virtually visible, the leg maybe placed closer to the real leg of the patient 504, but when thepatient 508 is viewed in full, this distance may be increased. Thevirtual representation of the remote surgical field may be based onimages (e.g., video) captured by a camera affixed to the remote roboticarm 506. For example, the camera on the remote robotic arm 506 mayidentify a feature, and another camera or an AR device in the remotesurgical field may be used to see different points of view (e.g., cameraviews).

In an example, the remote patient 508 is a live surgical patient and thelocal patient 504 is a live surgical patient. In this example, theremote patient 508 may be remotely operated on using the robotic arm 506by a surgeon in the real space of the display view 500. For example, thesurgeon may simultaneously operate on both the remote patient 508 andthe local patient 504. Simultaneously in this example may mean thesurgeon switches between the patients at various operations of thesurgery, such as at each step or after particular sequences of steps, orone surgery may be completed before the next is started, but bothpatients are available, viewable, or ready for surgerycontemporaneously. In this version of this example, the surgeon maycomplete surgeries more quickly because multiple staff, operating rooms,and surgical equipment may be used in parallel rather than requiringserial surgeries. In another version of this example, the remote patient508 may be operated on by a remote surgeon (e.g., with or without theuse of the robotic arm 506), and the surgeon in the local space of thedisplay view 500 may be called in to consult or provide assistance(e.g., with a portion of a procedure, such as operation of the remoterobotic arm 506, for example when the remote surgeon is less experiencedusing a robotic arm). The remote patient 508 is viewable for theconsultation (e.g., in real-time) such that the surgeon in the localspace may give direction or advise without needing to physically leavethe local surgical field. This version of the example may beparticularly useful when the remote surgeon is a student, a newersurgeon, or the surgery is occurring in a remote building, city,country, etc.

In an example, the remote patient 508 is a live surgical patient and thelocal patient 504 is a cadaver. In this example, a surgeon in localspace 500 may view a remote surgery, which may be occurring in real-timeor may have already occurred and is viewed on replay. This exampleallows for a student or newer surgeon to complete a procedure (e.g., anew type or particularly difficult type) on a cadaver while being ableto view a similar or the same procedure virtually. The virtualrepresentation may be viewed at different angles, zoomed, or the like.When the virtual representation is a replay, the surgery may bereversed, sped up, paused, etc. In another version of this example, aremote surgeon may request advice or support from the local surgeon, whomay attempt a portion of the surgery on the cadaver before the portionis attempted on the live remote patient 508. This allows for the portionof the procedure to be tested without damage to the live remote patient508.

In an example, the remote patient 508 is a cadaver and the local patient504 is a live surgical patient. In this example, a surgeon in the localspace of the display view 500 may attempt a portion of a procedure onthe remote cadaver before attempting the portion on the live localpatient 504. The local surgeon may control the remote robotic arm 506while performing the portion on the cadaver. The remote robotic arm 506may save actions undertaken during the operation, which may be sent tothe local robotic arm 502, and optionally edited. The saved actions maybe repeated by the local robotic arm 502, for example to perform anautonomous portion of the procedure that has been tested on the cadaver.Differences between the cadaver and the local live patient 504 may beused to alter the saved actions, for example by scaling, moving targetpoints, or the like. Differences in the robotic arms may be accountedfor based on a calibration step performed before starting the surgicalprocedure. In an example, a procedure may be tested on a cadaver usingthe remote robotic arm 506, then successful actions may be transferredto the local robotic arm 502 for autonomous action or force-resist typemovement by the local robotic arm 502 when performing the procedure onthe local patient 504.

In an example, the remote patient 508 is a cadaver and the local patient504 is a cadaver. In this example, a surgeon may practice a procedure ontwo different cadavers contemporaneously to identify differences inresults from changes to the procedure. In another version of thisexample, the surgeon may perform a procedure for a student or newersurgeon while the student or newer surgeon operates remotely on thecadaver. In this version, the local surgeon may view and optionallycritique the remote surgery. The remote surgical field may have asimilar setup, allowing the student or newer surgeon to view theteaching surgeon's operation in an augmented or virtual reality view.

In any of the above examples, more than one remote surgical field may bepresented. For example, a teaching surgeon may view multiple remotestudent surgeries. When two or more remote surgical fields arepresented, they may be scaled to fit in the display view 500. A remotesurgical field may be placed adjacent another remote surgical field, inan example.

A local surgeon may provide assistance when requested for a remoteprocedure, such as in a collaborative mode with the remote surgical arm506. The collaborative mode may allow the local surgeon to move theremote surgical arm 506, while allowing the remote surgeon to stop theremote surgical arm 506. In another example, the local surgeon may bestop or take over control of the remote surgical arm 506 whilemonitoring the remote surgeon operating with the remote surgical arm506. In yet another example, the local surgeon may control the localrobotic arm 502, which in turn may send information to control theremote robotic arm 506 or the local robotic arm 502 may move in responseto information received from the remote robotic arm 506. For example,the robotic arms may move in concert, such that either the remote orlocal surgeon may control the procedure. One of the surgeons may act toresist erroneous movements while the other of the surgeons performs theprocedure, each using their respective robotic arm. In an example, theremote surgical field may represent a surgical field in a same buildingas the local surgical field.

FIG. 6 illustrates a flowchart showing a technique 600 for displaying avirtual representation of a remote surgical field within a localsurgical field in accordance with some embodiments. The technique 600may be performed by a processor, for example by executing instructionsstored in memory. The technique 600 includes an operation 602 to receivea video stream of a remote surgical subject.

The technique 600 includes an operation 604 to present, using anaugmented reality display, within a surgical field, a virtual surgicalfield representing the remote surgical subject. Operation 604 mayinclude presenting the virtual surgical field while permitting a patientwithin the surgical field to be viewed through the augmented realitydevice. The virtual surgical field may be presented adjacent to thepatient, in an example. Adjacent may mean separated by a fixed distancein absolute space within the surgical field, for example a foot, a fewfeet, etc. In another example, adjacent may mean separated by a relativedistance as perceived through the augmented reality device (e.g.,appearing to be separated by a foot, a few feet, etc.). Adjacent maymean touching, or almost touching.

The remote surgical subject may include a patient in another operatingroom within a building also housing the surgical field, a cadaver, orthe like. The technique 600 may further include an operation to receivea voice instruction and send the voice instruction to a remote speaker(e.g., within a remote surgical field corresponding to and representedby the virtual surgical field). The technique 600 may include receivinga request to present the virtual surgical field before presenting thevirtual surgical field (e.g., from a colleague, student, etc.). Thevirtual surgical field may be used for testing aspects of a technique(e.g., with a cadaver), for helping or consulting on a case, or toperform an entire procedure, in various examples. A second virtualsurgical field may be presented (e.g., adjacent to the patient, such ason an opposite side, or adjacent to the first surgical field) forinteraction or observation of a second remote surgical subject.

The virtual surgical field may be displayed including a virtualrepresentation of a remote surgical robot. The remote surgical robot maybe controlled by a command issued within the surgical field, for examplevia a voice command, a gesture, a user input on a device, touchscreen,virtual indication, a written, typed, haptic command, or the like. Theremote surgical robot may be guided via a gesture. In an example, thevirtual surgical field may be displayed based on output of a cameraaffixed to an end effector of the remote surgical robot.

FIG. 7 illustrates a robot sterilization system 700 in accordance withsome embodiments. The robot sterilization system 700 includes a roboticarm 704, and a sterilization unit 706, which may be embedded in a base702 of the robotic arm 704 or may be separate from the robotic arm 704.When separate, the sterilization unit 706 may be mounted under therobotic arm 704 or affixed to a portion of the robotic arm 704 (e.g.,the base 702).

The sterilization unit 706 may include an opening 708 that may be usedto output an instrument (e.g., instrument 712). In an example, aninstrument may be output from the opening 708, for example using amechanism within the sterilization unit 706. In another example, thesterilization unit 706 may include a tray 710, which may be output fromthe opening 708, the tray 710 used to convey the instrument 712. In yetanother example, a door of the sterilization unit 706 may open to allowa user to remote an instrument. In still another example, the roboticarm 704 may be used to retrieve an instrument from within thesterilization unit 706. For example, the robotic arm 704 may retrieve aninstrument from within the sterilization unit 706 based on knownlocations of instruments within the sterilization unit 706.

A door may be used to reload the sterilization unit 706 in an example.The sterilization unit 706 may include a sterile environment without thecapability of sterilizing instruments. In this example, thesterilization unit 706 is a passive sterile storage unit. In anotherexample, the sterilization unit 706 may be used to sterilize aninstrument. In this example, the sterilization unit 706 may usesterilization equipment to sterilize the instrument, such as by usingultraviolet light, steam, gas, an autoclave, alcohol, heat pressure,glass beads, or the like.

The sterilization unit 706 may be controlled by a user interface orcontrol mechanism, such as one incorporated in the base 702 or one alsoused to control the robotic arm 704 (e.g., an augmented reality userinterface, a display screen, a microphone and algorithm for interpretingaudible commands, the robotic arm 704 itself, or the like). Controls mayinclude initiating sterilization of an instrument (or all instrumentswithin the sterilization unit 706) or outputting an instrument (e.g.,opening a door, outputting a specific selected instrument, outputting anext instrument in a procedure, or outputting a machine learning modelidentified instrument at a particular step in a procedure).

The instrument 712 may be output automatically, for example based onsurgeon preferences, a machine learned model, or the like. For example,image processing may be used to determine a step of a procedure that iscompleted or almost completed, and an instrument for a next step may beoutput. In another example, movement of the robotic arm 704 may be usedto determine that an instrument is needed and output that instrument. Inthis example, the movement may be a stored movement or a movement uniqueto a portion of a surgical procedure that identifies a next step.

FIG. 8 illustrates a flowchart showing a technique 800 for storing asterilized instrument using a robotic system in accordance with someembodiments. The technique 800 may be implemented using a surgicalrobotic system, such as with a processor.

The technique 800 includes an operation 802 to provide a sterileenvironment. The sterile environment may be housed by a sterilizationunit to store an instrument. The sterilization unit may be mounted underor form a base of a surgical robotic arm of the surgical robotic system.In the example where the sterilization unit is mounted under thesurgical robotic arm, the sterilization unit may be a portablesterilization unit. In an example, the sterilization unit may be asterile storage unit without sterilization capabilities itself. Inanother example, the sterilization unit may be configured to activelysterilize the instrument, for example using ultraviolet light, steam,gas, an autoclave, alcohol, heat pressure, glass beads, or the like. Thesterilization unit may store a plurality of instruments including theinstrument.

The technique 800 includes an operation 804 to determine whether theinstrument is needed. In response to a determination that the instrumentis not needed, the technique 800 may return to operation 602 or 604.Operation 804 may include using machine learning techniques to determinethat the instrument is needed. For example, a trained model (which mayinclude a binary classification, a regression model, a convolutionalneural network, etc.) may be used to determine that a surgical step hasbeen reached, that a time has passed, previously stored surgeonpreferences, probability, a selected workflow, or the like. In otherexamples, a command, such as a spoken command, a gesture, an interactionwith a physical or virtual user interface, or other techniques may beused to determine that the instrument is needed (e.g., a request for theinstrument).

The technique 800 includes an operation 806 to, in response to adetermination that the instrument is needed, provide access to theinstrument from the sterile environment. Operation 806 may includedisplaying an indication of the instrument using an augmented realitydisplay device. Operation 806 may include causing an enclosure of thesterilization unit to open, exposing the sterile environment includingthe instrument. Operation 806 may include causing the surgical roboticarm to retrieve the instrument. Operation 806 may include causing theinstrument to be output from the sterilization unit via a mechanicalconveyance. Operation 806 may include providing a set of sterileinstruments, including the instrument, for a procedure.

FIG. 9 illustrates a system 900 for surgical instrument identificationusing an augmented reality display in accordance with some embodiments.The system 900 may be used to perform any of the techniques 400 or 600described in relation to FIG. 4 or 6, for example, by using a processor902. The system 900 includes an augmented reality device 901 that may bein communication with a database 916. The augmented reality device 901includes a processor 902, memory 904, an AR display 908, and a camera906. The augmented reality device 901 may include a sensor 910, aspeaker 912, or a haptic controller 914. The database 916 may includeimage storage 918 or preoperative plan storage 920. In an example, theaugmented reality device 901 may be a HoloLens manufactured by Microsoftof Redmond, Wash.

The processor 902 of the augmented reality device 901 includes anaugmented reality modeler 903. The augmented reality modeler 903 may beused by the processor 902 to create the augmented reality environment.For example, the augmented reality modeler 903 may receive dimensions ofa room, such as from the camera 906 or sensor 910, and create theaugmented reality environment to fit within the physical structure ofthe room. In another example, physical objects may be present in theroom and the augmented reality modeler 903 may use the physical objectsto present virtual objects in the augmented reality environment. Forexample, the augmented reality modeler 903 may use or detect a tablepresent in the room and present a virtual object as resting on thetable. The AR display 908 may display the AR environment overlaid on areal environment. The display 908 may show a virtual object, using theAR device 901, such as in a fixed position in the AR environment. Theaugmented reality modeler 903 may receive a video stream of a remotesurgical field for virtually displaying within the room. In an example,a dimension of a virtual object (e.g., a remote surgical field) may bemodified (e.g., shrunk) to be virtually displayed within the room. In anexample, the augmented reality device 901 may provide a zoom function toallow a user to zoom in on a portion of a virtual object (e.g., within avirtual surgical field).

The augmented reality device 901 may include a sensor 910, such as aninfrared sensor. The camera 906 or the sensor 910 may be used to detectmovement, such as a gesture by a surgeon or other user, that may beinterpreted by the processor 902 as attempted or intended interaction bythe user with the virtual target. The processor 902 may identify anobject in a real environment, such as through processing informationreceived using the camera 906.

The AR display 908, for example during a surgical procedure, maypresent, such as within a surgical field while permitting the surgicalfield to be viewed through the augmented reality display, a virtualfeature corresponding to a physical feature hidden by an anatomicalaspect of a patient. The virtual feature may have a virtual position ororientation corresponding to a first physical position or orientation ofthe physical feature. In an example, the virtual position or orientationof the virtual feature may include an offset from the first physicalposition or orientation of the physical feature. The offset may includea predetermined distance from the augmented reality display, a relativedistance from the augmented reality display to the anatomical aspect, orthe like.

FIG. 10 illustrates generally an example of a block diagram of a machine1000 upon which any one or more of the techniques (e.g., methodologies)discussed herein may perform in accordance with some embodiments. Inalternative embodiments, the machine 1000 may operate as a standalonedevice or may be connected (e.g., networked) to other machines. In anetworked deployment, the machine 1000 may operate in the capacity of aserver machine, a client machine, or both in server-client networkenvironments. The machine 1000 may be a personal computer (PC), a tabletPC, a personal digital assistant (PDA), a mobile telephone, a webappliance, a network router, switch or bridge, or any machine capable ofexecuting instructions (sequential or otherwise) that specify actions tobe taken by that machine. Further, while only a single machine isillustrated, the term “machine” shall also be taken to include anycollection of machines that individually or jointly execute a set (ormultiple sets) of instructions to perform any one or more of themethodologies discussed herein, such as cloud computing, software as aservice (SaaS), other computer cluster configurations.

Examples, as described herein, may include, or may operate on, logic ora number of components, modules, or like mechanisms. Such mechanisms aretangible entities (e.g., hardware) capable of performing specifiedoperations when operating. In an example, the hardware may bespecifically configured to carry out a specific operation (e.g.,hardwired). In an example, the hardware may include configurableexecution units (e.g., transistors, circuits, etc.) and a computerreadable medium containing instructions, where the instructionsconfigure the execution units to carry out a specific operation when inoperation. The configuring may occur under the direction of theexecutions units or a loading mechanism. Accordingly, the executionunits are communicatively coupled to the computer readable medium whenthe device is operating. For example, under operation, the executionunits may be configured by a first set of instructions to implement afirst set of features at one point in time and reconfigured by a secondset of instructions to implement a second set of features.

Machine (e.g., computer system) 1000 may include a hardware processor1002 (e.g., a central processing unit (CPU), a graphics processing unit(GPU), a hardware processor core, or any combination thereof), a mainmemory 1004 and a static memory 1006, some or all of which maycommunicate with each other via an interlink (e.g., bus) 1008. Themachine 1000 may further include a display unit 1010, an alphanumericinput device 1012 (e.g., a keyboard), and a user interface (UI)navigation device 1014 (e.g., a mouse). In an example, the display unit1010, alphanumeric input device 1012 and UI navigation device 1014 maybe a touch screen display. The display unit 1010 may include goggles,glasses, or other AR or VR display components. For example, the displayunit may be worn on a head of a user and may provide a heads-up-displayto the user. The alphanumeric input device 1012 may include a virtualkeyboard (e.g., a keyboard displayed virtually in a VR or AR setting.

The machine 1000 may additionally include a storage device (e.g., driveunit) 1016, a signal generation device 1018 (e.g., a speaker), a networkinterface device 1020, and one or more sensors 1021, such as a globalpositioning system (GPS) sensor, compass, accelerometer, or othersensor. The machine 1000 may include an output controller 1028, such asa serial (e.g., universal serial bus (USB), parallel, or other wired orwireless (e.g., infrared (IR), near field communication (NFC), etc.)connection to communicate or control one or more peripheral devices.

The storage device 1016 may include a machine readable medium 1022 thatis non-transitory on which is stored one or more sets of data structuresor instructions 1024 (e.g., software) embodying or utilized by any oneor more of the techniques or functions described herein. Theinstructions 1024 may also reside, completely or at least partially,within the main memory 1004, within static memory 1006, or within thehardware processor 1002 during execution thereof by the machine 1000. Inan example, one or any combination of the hardware processor 1002, themain memory 1004, the static memory 1006, or the storage device 1016 mayconstitute machine readable media.

While the machine readable medium 1022 is illustrated as a singlemedium, the term “machine readable medium” may include a single mediumor multiple media (e.g., a centralized or distributed database, orassociated caches and servers) configured to store the one or moreinstructions 1024.

The term “machine readable medium” may include any medium that iscapable of storing, encoding, or carrying instructions for execution bythe machine 1000 and that cause the machine 1000 to perform any one ormore of the techniques of the present disclosure, or that is capable ofstoring, encoding or carrying data structures used by or associated withsuch instructions. Non-limiting machine readable medium examples mayinclude solid-state memories, and optical and magnetic media. Specificexamples of machine readable media may include: non-volatile memory,such as semiconductor memory devices (e.g., Electrically ProgrammableRead-Only Memory (EPROM), Electrically Erasable Programmable Read-OnlyMemory (EEPROM)) and flash memory devices; magnetic disks, such asinternal hard disks and removable disks; magneto-optical disks; andCD-ROM and DVD-ROM disks.

The instructions 1024 may further be transmitted or received over acommunications network 1026 using a transmission medium via the networkinterface device 1020 utilizing any one of a number of transferprotocols (e.g., frame relay, internet protocol (IP), transmissioncontrol protocol (TCP), user datagram protocol (UDP), hypertext transferprotocol (HTTP), etc.). Example communication networks may include alocal area network (LAN), a wide area network (WAN), a packet datanetwork (e.g., the Internet), mobile telephone networks (e.g., cellularnetworks), Plain Old Telephone (POTS) networks, and wireless datanetworks (e.g., Institute of Electrical and Electronics Engineers (IEEE)802.11 family of standards known as Wi-Fi®, as the personal area networkfamily of standards known as Bluetooth® that are promulgated by theBluetooth Special Interest Group, peer-to-peer (P2P) networks, amongothers. In an example, the network interface device 1020 may include oneor more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or oneor more antennas to connect to the communications network 1026. In anexample, the network interface device 1020 may include a plurality ofantennas to wirelessly communicate using at least one of single-inputmultiple-output (SIMO), multiple-input multiple-output (MIMO), ormultiple-input single-output (MISO) techniques. The term “transmissionmedium” shall be taken to include any intangible medium that is capableof storing, encoding or carrying instructions for execution by themachine 1000, and includes digital or analog communications signals orother intangible medium to facilitate communication of such software.

FIGS. 11A-11B illustrate user interface components for landmark planningand plan evaluation in accordance with some embodiments.

FIG. 11A illustrates user interface feedback components 1102-1108 toprovide feedback based on location of a patient, an instrument, or aplaced or planned landmark. The feedback components 1102-1108 may bepresented on a user interface on a display, such as a display screen, anaugmented reality view, or a virtual reality view. For example, when alandmark is placed by a surgeon, a feedback component may pop up in anaugmented reality display so that the surgeon may view the feedbackcomponent.

Feedback component 1102 may indicate that movement is needed (e.g., of abone or patient, of an instrument, of a camera, etc.), such as beforestarting a landmark placement process. Feedback component 1104 mayindicate that an acquisition point is too close to a previous landmark(e.g., virtual or placed). Feedback component 1106 may indicate that anacquisition point was outside a target area (e.g., an error occurred, astray landmark was placed, or a landmark was accidentally placed outsidea bone or other target area). Feedback component 1108 may indicate thata point that was placed is not aligned with a planned or virtuallandmark point. The feedback component 1108 may appear instead of or inaddition to a visual indication on the landmark itself (e.g., using anAR display).

The feedback component 1102-1108 may be interactable components, such asin an AR display. In this example, a feedback component may be selectedto provide further visual feedback, such as a video example,highlighting of a landmark (virtual, placed, or planned), an arrow orother indicator pointing out the issue, or the like.

FIG. 11B illustrates example virtual landmarks displayed for example ina user interface component of an AR display. In this example, acomponent 1110 illustrates a portion of a knee bone with various virtuallandmarks 1112-1116 displayed. In an example, the virtual landmarks1112-1116 may be visually distinct, such as including a color coding, ananimation (e.g., flashing), have a transparency, or the like. In anexample, virtual landmark 1112 may indicate an upcoming landmark pointto be placed, virtual landmark 1114 may indicate a current landmarkpoint being placed, and virtual landmark 1116 may indicate a successfullandmark point previously placed.

The virtual landmarks 1112-1116 may be labeled in some examples, such aswith a name for the landmark (e.g., femoral canal entry, posteriorcondyles, etc.). The labels may be displayed in the AR displayvirtually, and in some examples may be hidden or revealed according touser preference or selection. The virtual landmarks 1112-1116 may beremoved when completed (e.g., for successfully placed landmarks), suchas automatically or after user confirmation. In some examples, aplurality of virtual landmarks may be displayed representing allsuccessfully placed points or all points that still need to becompleted.

In the examples of FIGS. 11A-11B, warnings, alerts, or information maybe presented in an AR display during landmark selection. For example,when bone registration is not successful, an alert or warning may bedisplayed that some landmarks may have been placed incorrectly. Theselandmarks may be highlighted or otherwise indicated in the AR display(e.g., as a virtual landmark such as 1112). Other information may bedisplayed during the landmarking process, such as surgeon notes forexample notes captured during planning and placement of virtuallandmarks, suggestions, steps, or the like.

Example 1 is a method for using an augmented reality device in asurgical field comprising: receiving, at a processor, an indication of alocation of a landmark on a bone of a patient; retrieving, using theprocessor, a planned location of the landmark on the bone of the patientbased on a pre-operative image of the bone of the patient; presenting,using an augmented reality display, within a surgical field, whilepermitting the surgical field to be viewed through the augmented realitydisplay, a virtual indication of the landmark at the location and avirtual indication of the landmark at the planned location; andreceiving, at the processor, a response to a request for confirmation ofthe location of the landmark.

In Example 2, the subject matter of Example 1 includes, wherein theindication of the location of the landmark is stored in a database.

In Example 3, the subject matter of Examples 1-2 includes, wherein theindication of the location of the landmark is received directly from alandmark generation device.

In Example 4, the subject matter of Examples 1-3 includes, wherein theresponse confirms the location of the landmark.

In Example 5, the subject matter of Examples 1-4 includes, wherein theresponse changes the location of the landmark to the planned location.

In Example 6, the subject matter of Examples 1-5 includes, wherein theresponse includes a new location for the landmark.

In Example 7, the subject matter of Examples 1-6 includes, removing thevirtual indication in response to receiving the response.

In Example 8, the subject matter of Examples 1-7 includes, registeringthe bone using a 3D model before receiving the indication of thelandmark.

In Example 9, the subject matter of Examples 1-8 includes, receiving anindication to move a virtual navigation menu presented in the augmentedreality display.

In Example 10, the subject matter of Examples 1-9 includes, wherein aposition and orientation of the bone is determined using bone trackingvia a passive robotic arm.

In Example 11, the subject matter of Examples 1-10 includes, displayinga live video, using the augmented reality display, of the bone using acamera affixed to an end effector of a robotic arm.

Example 12 is a system configured to perform operations of any of any ofthe methods of Examples 1-11.

Example 13 is at least one machine-readable medium includinginstructions for operation of a computing system, which when executed bya machine, cause the machine to perform operations of any of the methodsof Examples 1-11.

Example 14 is an apparatus comprising means for performing any of themethods of Examples 1-11.

Example 15 is a method for using an augmented reality device in asurgical field comprising: receiving a video stream of a remote surgicalsubject; and presenting, using an augmented reality display, within asurgical field, while permitting a patient within the surgical field tobe viewed through the augmented reality display, a virtual surgicalfield adjacent to the patient, the virtual surgical field representingthe remote surgical subject.

In Example 16, the subject matter of Example 15 includes, receiving avoice instruction and sending the voice instruction to a remote speaker.

In Example 17, the subject matter of Examples 15-16 includes, whereinthe remote surgical subject includes a patient in another operating roomwithin a building housing the surgical field.

In Example 18, the subject matter of Examples 15-17 includes, whereinthe remote surgical subject includes a cadaver.

In Example 19, the subject matter of Examples 15-18 includes, whereinpresenting the virtual surgical field includes displaying a virtualrepresentation of a remote surgical robot.

In Example 20, the subject matter of Example 19 includes, sending acommand to the remote surgical robot.

In Example 21, the subject matter of Example 20 includes, wherein thecommand includes a written, typed, touchscreen-selected, augmentedreality selected, or spoken command.

In Example 22, the subject matter of Examples 19-21 includes, guidingthe remote surgical robot via a gesture.

In Example 23, the subject matter of Examples 19-22 includes, displayinga view of the virtual surgical field using a camera affixed to an endeffector of the remote surgical robot.

In Example 24, the subject matter of Examples 15-23 includes, receivinga request to present the virtual surgical field before presenting thevirtual surgical field.

In Example 25, the subject matter of Examples 15-24 includes, presentinga second virtual surgical field adjacent to the patient or adjacent tothe virtual surgical field, the second virtual surgical fieldrepresenting a second remote surgical subject.

Example 26 is a system configured to perform operations of any of any ofthe methods of Examples 15-25.

Example 27 is at least one machine-readable medium includinginstructions for operation of a computing system, which when executed bya machine, cause the machine to perform operations of any of the methodsof Examples 15-25.

Example 28 is an apparatus comprising means for performing any of themethods of Examples 15-25.

Example 29 is a surgical robotic system comprising: a surgical roboticarm; a sterilization unit enclosing a sterile environment and storing aninstrument; a processor configured to: determine that the instrument isneeded in an upcoming portion of a surgical procedure; and provideaccess to the instrument.

In Example 30, the subject matter of Example 29 includes, wherein thesterilization unit is a base of the surgical robotic arm.

In Example 31, the subject matter of Examples 29-30 includes, whereinthe sterilization unit is a portable sterilization unit, and wherein thesurgical robotic arm is configured to be mounted on the portablesterilization unit.

In Example 32, the subject matter of Examples 29-31 includes, whereinthe sterilization unit is a sterile storage unit without sterilizationcapabilities.

In Example 33, the subject matter of Examples 29-32 includes, whereinthe sterilization unit is configured to actively sterilize theinstrument.

In Example 34, the subject matter of Examples 29-33 includes, whereinthe sterilization unit is configured to store a plurality of instrumentsincluding the instrument.

In Example 35, the subject matter of Examples 29-34 includes, whereinthe determination that the instrument is needed is based on machinelearning.

In Example 36, the subject matter of Examples 29-35 includes, whereinthe determination that the instrument is needed is based on a previouslystored surgeon preference.

In Example 37, the subject matter of Examples 29-36 includes, whereinthe determination that the instrument is needed is based on aprobability using a selected workflow and a timer.

In Example 38, the subject matter of Examples 29-37 includes, whereinthe determination that the instrument is needed includes receiving arequest for the instrument, including at least one of a spoken command,a touch on a touchscreen, an interaction with an augmented reality userinterface, or a gesture.

In Example 39, the subject matter of Examples 29-38 includes, wherein toprovide access to the instrument, the processor is further configured todisplay an indication of the instrument using an augmented realitydisplay device.

In Example 40, the subject matter of Examples 29-39 includes, wherein toprovide access to the instrument, the processor is further configured tocause an enclosure of the sterilization unit to open, exposing thesterile environment including the instrument.

In Example 41, the subject matter of Examples 29-40 includes, wherein toprovide access to the instrument, the processor is further configured tocause the surgical robotic arm to retrieve the instrument.

In Example 42, the subject matter of Examples 29-41 includes, wherein toprovide access to the instrument, the processor is further configured tocause the instrument to be output from the sterilization unit via amechanical conveyance.

In Example 43, the subject matter of Examples 29-42 includes, wherein toprovide access to the instrument, the processor is further configured toprovide a set of sterile instruments, including the instrument, for aprocedure.

Example 44 is a system configured to perform operations of any of any ofthe methods of Examples 29-43.

Example 45 is at least one machine-readable medium includinginstructions for operation of a computing system, which when executed bya machine, cause the machine to perform operations of any of the methodsof Examples 29-43.

Example 46 is an apparatus comprising means for performing any of themethods of Examples 29-43.

Example 47 is a method of using a surgical robotic system comprising:determining that an instrument, stored in a sterilization unit enclosinga sterile environment is needed in an upcoming portion of a surgicalprocedure; and providing access to the instrument from the sterilizationunit that is mounted under or forms a base of a surgical robotic arm ofthe surgical robotic system.

Example 48 is a method for using an augmented reality device in asurgical field comprising: receiving, at a processor, an indication of alocation of a landmark on a bone of a patient; retrieving, using theprocessor, a planned location of the landmark on the bone of the patientbased on a pre-operative image of the bone of the patient; presentingwithin the surgical field, using an augmented reality display of theaugmented reality device, while permitting the surgical field to beviewed through the augmented reality display, a virtual indication ofthe landmark at the location and a virtual indication of the landmark atthe planned location; and receiving, at the processor, a confirmation ofthe location of the landmark.

In Example 49, the subject matter of Example 48 includes, wherein theindication of the location of the landmark is stored in a database.

In Example 50, the subject matter of Examples 48-49 includes, whereinthe indication of the location of the landmark is received directly froma landmark generation device.

In Example 51, the subject matter of Examples 48-50 includes, whereinthe confirmation indicates the location of the landmark is correct.

In Example 52, the subject matter of Examples 48-51 includes, for asecond landmark having a second location and a second planned location,receiving a change for the second landmark from the second location tothe second planned location, and outputting information corresponding tothe change to the augmented reality display.

In Example 53, the subject matter of Examples 48-52 includes, for asecond landmark having a second location and a second planned location,receiving a change for the second landmark from the second location to anew location other than the second location and the second plannedlocation, and outputting information corresponding to the new locationto the augmented reality display for presenting.

In Example 54, the subject matter of Examples 48-53 includes, removingthe virtual indications in response to receiving the confirmation.

In Example 55, the subject matter of Examples 48-54 includes, D modelbefore receiving the indication of the landmark.

In Example 56, the subject matter of Examples 48-55 includes, receivingan indication to move a virtual navigation menu presented in theaugmented reality display.

In Example 57, the subject matter of Examples 48-56 includes, wherein aposition and orientation of the bone is determined using bone trackingvia a passive robotic arm.

In Example 58, the subject matter of Examples 48-57 includes, displayinga live video, using the augmented reality display, of the bone using acamera affixed to an end effector of a robotic arm.

Example 59 is an augmented reality device in a surgical fieldcomprising: a processor; memory including instructions, which whenexecuted by the processor, cause the processor to perform operations to:receive an indication of a location of a landmark on a bone of apatient; retrieve a planned location of the landmark on the bone of thepatient based on a pre-operative image of the bone of the patient; andreceive a confirmation of the location of the landmark; and an augmentedreality display to, before the processor receives the confirmation,present within the surgical field while permitting the surgical field tobe viewed through the augmented reality display, a virtual indication ofthe landmark at the location and a virtual indication of the landmark atthe planned location.

In Example 60, the subject matter of Example 59 includes, wherein theindication of the location of the landmark is stored in a database.

In Example 61, the subject matter of Examples 59-60 includes, whereinthe indication of the location of the landmark is received directly froma landmark generation device.

In Example 62, the subject matter of Examples 59-61 includes, whereinthe confirmation indicates the location of the landmark is correct.

In Example 63, the subject matter of Examples 59-62 includes, whereinthe instructions further cause the processor to, for a second landmarkhaving a second location and a second planned location, receive a changefor the second landmark from the second location to the second plannedlocation, and output information corresponding to the change to theaugmented reality display for presenting.

In Example 64, the subject matter of Examples 59-63 includes, whereinthe instructions further cause the processor to, for a second landmarkhaving a second location and a second planned location, receive a changefor the second landmark from the second location to a new location otherthan the second location and the second planned location, and outputinformation corresponding to the new location to the augmented realitydisplay for presenting.

In Example 65, the subject matter of Examples 59-64 includes, whereinthe instructions further cause the processor to remove the virtualindications in response to receiving the confirmation.

In Example 66, the subject matter of Examples 59-65 includes, D modelbefore receiving the indication of the landmark.

Example 67 is at least one machine-readable medium includinginstructions for operating an augmented reality device in a surgicalfield, which when executed by a processor, cause the processor toperform operations to: retrieving a plurality of planned locationscorresponding to each of a plurality of landmarks on a bone of a patientbased on pre-operative imaging of the bone of the patient; presentingwithin the surgical field, using an augmented reality display of theaugmented reality device, while permitting the surgical field to beviewed through the augmented reality display, virtual indications of theplurality of landmarks at the plurality of planned locations; andreceiving a confirmation of a first planned location as presented usingthe augmented reality display for a first landmark of the plurality oflandmarks; and receiving a change to a second planned location aspresented using the augmented reality display for a second landmark ofthe plurality of landmarks.

Example 68 is at least one machine-readable medium includinginstructions that, when executed by processing circuitry, cause theprocessing circuitry to perform operations to implement of any ofExamples 1-67.

Example 69 is an apparatus comprising means to implement of any ofExamples 1-67.

Example 70 is a system to implement of any of Examples 1-67.

Example 71 is a method to implement of any of Examples 1-67.

Method examples described herein may be machine or computer-implementedat least in part. Some examples may include a computer-readable mediumor machine-readable medium encoded with instructions operable toconfigure an electronic device to perform methods as described in theabove examples. An implementation of such methods may include code, suchas microcode, assembly language code, a higher-level language code, orthe like. Such code may include computer readable instructions forperforming various methods. The code may form portions of computerprogram products. Further, in an example, the code may be tangiblystored on one or more volatile, non-transitory, or non-volatile tangiblecomputer-readable media, such as during execution or at other times.Examples of these tangible computer-readable media may include, but arenot limited to, hard disks, removable magnetic disks, removable opticaldisks (e.g., compact disks and digital video disks), magnetic cassettes,memory cards or sticks, random access memories (RAMs), read onlymemories (ROMs), and the like.

What is claimed is:
 1. A method for using an augmented reality device ina surgical field comprising: receiving, at a processor, an indication ofa location of a landmark on a bone of a patient; retrieving, using theprocessor, a planned location of the landmark on the bone of the patientbased on a pre-operative image of the bone of the patient; presentingwithin the surgical field, using an augmented reality display of theaugmented reality device, while permitting the surgical field to beviewed through the augmented reality display, a virtual indication ofthe landmark at the location and a virtual indication of the landmark atthe planned location; and receiving, at the processor, a confirmation ofthe location of the landmark.
 2. The method of claim 1, wherein theindication of the location of the landmark is stored in a database. 3.The method of claim 1, wherein the indication of the location of thelandmark is received directly from a landmark generation device.
 4. Themethod of claim 1, wherein the confirmation indicates the location ofthe landmark is correct.
 5. The method of claim 1, further comprising,for a second landmark having a second location and a second plannedlocation, receiving a change for the second landmark from the secondlocation to the second planned location, and outputting informationcorresponding to the change to the augmented reality display.
 6. Themethod of claim 1, further comprising, for a second landmark having asecond location and a second planned location, receiving a change forthe second landmark from the second location to a new location otherthan the second location and the second planned location, and outputtinginformation corresponding to the new location to the augmented realitydisplay for presenting.
 7. The method of claim 1, further comprisingremoving the virtual indications in response to receiving theconfirmation.
 8. The method of claim 1, further comprising registeringthe bone using a 3D model before receiving the indication of thelandmark.
 9. The method of claim 1, further comprising receiving anindication to move a virtual navigation menu presented in the augmentedreality display.
 10. The method of claim 1, wherein a position andorientation of the bone is determined using bone tracking via a passiverobotic arm.
 11. The method of claim 1, further comprising displaying alive video, using the augmented reality display, of the bone using acamera affixed to an end effector of a robotic arm.
 12. An augmentedreality device in a surgical field comprising: a processor; memoryincluding instructions, which when executed by the processor, cause theprocessor to perform operations to: receive an indication of a locationof a landmark on a bone of a patient; retrieve a planned location of thelandmark on the bone of the patient based on a pre-operative image ofthe bone of the patient; and receive a confirmation of the location ofthe landmark; and an augmented reality display to, before the processorreceives the confirmation, present within the surgical field whilepermitting the surgical field to be viewed through the augmented realitydisplay, a virtual indication of the landmark at the location and avirtual indication of the landmark at the planned location.
 13. Theaugmented reality device of claim 12, wherein the indication of thelocation of the landmark is stored in a database.
 14. The augmentedreality device of claim 12, wherein the indication of the location ofthe landmark is received directly from a landmark generation device. 15.The augmented reality device of claim 12, wherein the confirmationindicates the location of the landmark is correct.
 16. The augmentedreality device of claim 12, wherein the instructions further cause theprocessor to, for a second landmark having a second location and asecond planned location, receive a change for the second landmark fromthe second location to the second planned location, and outputinformation corresponding to the change to the augmented reality displayfor presenting.
 17. The augmented reality device of claim 12, whereinthe instructions further cause the processor to, for a second landmarkhaving a second location and a second planned location, receive a changefor the second landmark from the second location to a new location otherthan the second location and the second planned location, and outputinformation corresponding to the new location to the augmented realitydisplay for presenting.
 18. The augmented reality device of claim 12,wherein the instructions further cause the processor to remove thevirtual indications in response to receiving the confirmation.
 19. Theaugmented reality device of claim 12, wherein the instructions furthercause the processor to register the bone using a 3D model beforereceiving the indication of the landmark.
 20. At least onemachine-readable medium including instructions for operating anaugmented reality device in a surgical field, which when executed by aprocessor, cause the processor to perform operations to: retrieving aplurality of planned locations corresponding to each of a plurality oflandmarks on a bone of a patient based on pre-operative imaging of thebone of the patient; presenting within the surgical field, using anaugmented reality display of the augmented reality device, whilepermitting the surgical field to be viewed through the augmented realitydisplay, virtual indications of the plurality of landmarks at theplurality of planned locations; and receiving a confirmation of a firstplanned location as presented using the augmented reality display for afirst landmark of the plurality of landmarks; and receiving a change toa second planned location as presented using the augmented realitydisplay for a second landmark of the plurality of landmarks.